Hierarchical reinforcement learning with subpolicies specializing for learned subgoals
نویسندگان
چکیده
This paper describes a method for hierarchical reinforcement learning in which high-level policies automatically discover subgoals, and low-level policies learn to specialize for different subgoals. Subgoals are represented as desired abstract observations which cluster raw input data. High-level value functions cover the state space at a coarse level; low-level value functions cover only parts of the state space at a fine-grained level. An experiment shows that this method outperforms several flat reinforcement learning methods. A second experiment shows how problems of partial observability due to observation abstraction can be overcome using high-level policies with memory.
منابع مشابه
Subgoal Discovery for Hierarchical Reinforcement Learning Using Learned Policies
Reinforcement learning addresses the problem of learning to select actions in order to maximize an agent’s performance in unknown environments. To scale reinforcement learning to complex real-world tasks, agent must be able to discover hierarchical structures within their learning and control systems. This paper presents a method by which a reinforcement learning agent can discover subgoals wit...
متن کاملAutonomous Subgoal Discovery and Hierarchical Abstraction for Reinforcement Learning Using Monte Carlo Method
Autonomous systems are often difficult to program. Reinforcement learning (RL) is an attractive alternative, as it allows the agent to learn behavior on the basis of sparse, delayed reward signals provided only when the agent reaches desired goals. However, standard reinforcement learning methods do not scale well for larger, more complex tasks. One promising approach to scaling up RL is hierar...
متن کاملAccelerating flat reinforcement learning on a robot by using subgoals in a hierarchical framework
Learning a motor skill task with Reinforcement Learning still takes a long time. A way to speed up the learning process without using much prior knowledge is to use subgoals. In this study, the use of subgoals decreased the learning time by a factor nine and we show that tests on a real robot give similar results. The price to be paid, in case the subgoals do not lie on the optimal path, is a w...
متن کاملHierarchical Reinforcement Learning Based Self-balancing Algorithm for Two-wheeled Robots
Abstract: Self-balancing control is the basis for applications of two-wheeled robots. In order to improve the self-balancing of twowheeled robots, we propose a hierarchical reinforcement learning algorithm for controlling the balance of two-wheeled robots. After describing the subgoals of hierarchical reinforcement learning, we extract features for subgoals, define a feature value vector and it...
متن کاملHierarchical Explanation-Based Reinforcement Learning
Explanation-Based Reinforcement Learning (EBRL) was introduced by Dietterich and Flann as a way of combining the ability of Reinforcement Learning (RL) to learn optimal plans with the generalization ability of Explanation-Based Learning (EBL) (Di-etterich & Flann, 1995). We extend this work to domains where the agent must order and achieve a sequence of subgoals in an optimal fashion. Hierarchi...
متن کامل